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FOREWORD 


This  technical  note  describes  work  conducted  as  part  of  the  Navy  Personnel  Research  and 
Development  Center’s  Communication  Networks  in  Training  (CNIT)  project  in  the  general  area  of 
remote-site  training.  The  CNIT  project  is  one  part  of  the  Schoolhouse  Training  product  line  and 
falls  under  the  Personnel  and  Training  Technology  (NP2A)  Block  of  the  6.2  Mission  Support 
Technology  Program  Element  06G2233N  (Work  Unit  RM33T23.02).  The  work  was  performed 
under  the  sponsorship  of  the  Office  of  Naval  Technology.  The  objective  of  the  project  is  to  find 
more  cost-effective  ways  to  train  personnel  who  are  geographically  remote  from  training 
resources.  The  project  has  been  exploring  the  use  of  new  communication  technologies  to  export 
training  to  geographically-remote  students.  Among  these  technologies  are  computer  networking, 
instructional  TV,  videotape,  audiographics,  videographics,  and  other  media.  This  technical  note 
describes  a  laboratory  study  involving  215  Navy  students  which  investigated  the  feasib  ly  of 
using  videoteletraining  to  deliver  hands-on  training.  The  findings  have  direct  implications  for  the 
design  of  future  distance  education  systems  in  the  Navy  and  elsewhere. 

The  recommendations  in  this  technical  note  are  intended  for  use  by  the  Chief  of  Naval 
Education  and  Training  and  Chief  of  Naval  Operations  (OP-11)  in  developing  policy  for  the 
application  of  advanced  communication  technology  in  the  Navy. 


J.  C.  MCLACHLAN 
Director,  Training  Research  Department 
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SUMMARY 


Problem  and  Background 

A  requirement  exists  to  train  Navy  personnel  who  are  geographically  remote  from  training 
resources.  Previous  research  and  development  work  has  demonstrated  that  videoteletraining  (VTT) 
can  be  an  effective  and  cost-effective  method  to  deliver  training  electronically  to  remote  Navy 
personnel.  To  date,  VTT  has  been  limited  to  the  delivery  of  lecture-based  training.  The  utility  of 
VTT  would  increase  if  it  could  be  used  to  deliver  hands-on  training. 

Objective 

The  primary  objective  of  the  project  is  to  find  more  cost-effective  ways  to  train  personnel  who 
are  geographically  remote  from  training  resources.  The  objective  of  the  work  described  in  this 
technical  note  was  to  test  the  feasibility  of  using  VTT  to  deliver  hands-on  training. 

Method 

The  method  included  steps  to  develop  a  research  plan,  prepare  for  VTT,  and  collect  and  analyze 
data. 

Results 

VTT  was  effective  for  lecture,  discussion,  and  hands-on  demonstration  portions  of  training  as 
indicated  by  the  final  examination,  student  course  evaluations,  and  observations. 

VTT  classroom  design  was  effective  for  hands-on  training  and  may  serve  as  a  model  for  use  by 
others  designing  VTT  classrooms  for  hands-on  training. 

The  most  difficult  aspect  of  hands-on  VTT  is  the  laboratory,  during  which  the  instructor 
typically  works  closely  with  students  to  demonstrate  procedures,  supervise,  assure  safety,  and 
certify  student  performance.  Two  different  strategies  for  handling  remote  laboratories  were  tried 
during  the  study;  (1)  having  students  view  videotapes  of  laboratory  procedures  instead  of 
participating  in  a  laboratory  and  (2)  having  a  facilitator  (instructor  surrogate)  conduct  the 
laboratories  off-line.  Students  who  observed  videotapes  instead  of  participating  in  a  laboratory 
took  longer  to  perform  on  all  performance  test  tasks  and  performed  less  accurately  on  two  out  of 
three  tasks.  Viewing  a  videotape  is  no  substitute  for  participating  in  a  laboratory,  though  it  is 
probably  better  than  no  laboratory  at  all.  The  research  provided  some  evidence  that  videotapes  are 
more  effective  than  live  demonstrations  in  helping  students  leant  to  perform  procedural  steps  that 
do  not  in  themselves  provide  feedback  on  their  correctness. 

The  second  laboratory  strategy  was  more  successful.  Observations'  indicated  that  the  learning 
processes  occurring  in  remote  laboratories  were  very  similar  to  those  of  a  live  class.  Additional 
research  is  required  to  determine  whether  or  not  this  strategy  would  be  generally  effective  for  a 
variety  of  content  areas. 


Recommendations 

1.  The  Chief  of  Naval  Education  and  Training  should  ensure  that  VTT  classrooms  used  to 
originate  courses  with  a  significant  hands-on  training  component  are  designed  to  satisfy  the  special 
requirements  of  these  courses.  Among  the  requirements  are  additional  storage  space  for  training 
aids,  an  open  and  flexible  floor  plan,  a  demonstration  area  covered  by  a  separate  camera,  an  in¬ 
structor’s  workstation,  and  a  cordless  microphone  to  provide  the  instructor  with  mobility. 

2.  The  Chief  of  Naval  Education  and  Training  should  employ  the  most  practical  and  effective 
strategy  known  at  this  time  for  conducting  hands-on  laboratories  at  remote  sites:  equip  remote  sites 
with  laboratory  equipment  and  have  a  facilitator  conduct  the  laboratories  off  line.  While  this  labo¬ 
ratory  strategy  is  workable,  other  less  costly  strategies  should  be  investigated  further. 

3.  The  Chief  of  Naval  Education  and  Training  should  originate  a  Problem  Description  and 
Need  Justification  for  research  to  test  and  evaluate  alternative  strategies  for  conducting  VTT 
hands-on  laboratories.  Among  these  strategies  are  remote  presence,  the  use  of  on-board  training 
packages  to  allow  supervisors  to  train  students  on  the  job,  and  other  strategies  that  may  be  feasible 
and  effective. 
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INTRODUCTION 


Problem  and  Background 

A  requirement  exists  to  train  Navy  personnel  who  are  geographically  remote  from  training 
resources.  Previous  research  and  development  work  has  demonstrated  that  videoteletraining  (VTT) 
can  be  an  effective  and  cost-effective  method  to  deliver  training  electronically  to  remote  Navy 
personnel  (Rupinski,  1991;  Rupinski  &  Stoloff,  1990;  Simpson,  Pugh,  &  Parchman,  1990,  1991; 
Stoloff,  1991).  To  date,  VTT  has  been  limited  to  the  delivery  of  lecture-based  training.  Much  of 
Navy  training  requires  students  to  develop  hands-on  skills  (e.g.,  to  disassemble,  adjust,  calibrate, 
or  otherwise  manipulate  machinery,  equipment,  devices,  or  other  physical  objects  using  tools  or 
test  equipment).  The  utility  of  VTT  would  increase  if  it  could  be  used  to  deliver  hands-on  training. 

Objective 

The  primary  objective  of  the  Communication  Network  in  Training  tCNTT)  project  is  to  find 
more  cost-effective  ways  to  train  personnel  who  are  geographically  remote  from  training 
resources.  To  meet  this  objective,  the  project  has  conducted  a  VTT  demonstration  project;  a  field 
survey  of  VTT  systems  in  public  education,  industry,  and  the  military;  and  a  series  of  laboratory 
studies  of  variables  influencing  VTT  user  acceptance  and  training  effectiveness.  This  project  work 
is  reported  in  Simpson  et  al.  (1990, 1991)  and  Pugh,  Parchman,  and  Simpson  (1991).  The  objective 
of  the  work  described  in  this  technical  note  was  to  lest  the  feasibility  of  using  VTT  to  deliver  hands- 
on  training. 

Research  Issues 

Instructional  TV  (ITV)  systems  have  traditionally  been  used  to  deliver  lecture-based  training. 
There  have  been  few  attempts  to  use  ITV  for  hands-on  training.  Conducting  lecture-based  training 
with  VTT  requires  an  redactor  to  stand  before  a  camera,  speak  into  a  microphone,  and  present 
visual  aids  using  some  form  of  an  extra  or  easel  camera.  Instructor-student  interaction  consists  of 
questions,  comments,  discussion,  and  other  verbal  discourse  picked  up  with  microphones  and 
projected  with  public  address  systems.  These  processes  are  relatively  straightforward. 

Courses  involving  hands-on  training  usually  include  lectures  and  discussion  so  they  require 
ever;  thing  that  lecture-based  VTT  docs.  In  addition,  hands-on  courses  generally  include 
dem  restrations  with  training  aids  of  varying  complexity  and  hands-on  laboratory  sessions  during 
which  the  instructor  works  closely  with  students  to  (1)  demonstrate  procedures,  (2)  supervise 
students,  (3)  assure  safety,  and  (4)  certify  student  performance.  Each  of  these  additional 
requirements  poses  a  challenge  to  VTT.  First,  the  classroom  must  be  adapted  to  permit  effective 
demonstrations,  or  alternative  methods  of  delivering  demonstrations  must  be  developed.  Second, 
alternative  methods  must  be  developed  to  handle  laboratories.  In  this  connection,  several  strategics 
are  possible.  The  simplest  strategy  is  to  eliminate  the  laboratory  from  training  at  remote  sites.  On 
its  face,  this  is  undesirable  in  terms  of  student  skill  mastery  and  would  probably  be  unacceptable 
to  the  Navy  training  establishment 
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A  second  strategy  is  to  equip  remote  sites  with  additional  audio  and  video  equipment  to  provide 
the  instructor  with  a  sort  of  remote  presence.  This  is  probably  not  practical  with  present  technology. 
Moreover,  an  instructor  who  was  not  physically  present  could  not  guarantee  student  safety. 

A  third  strategy  is  to  have  a  facilitator  conduct  the  laboratories  off-line.  This  strategy  is 
practical  to  the  extent  that  the  Navy  can  afford  to  equip  remote  classes  with  the  necessary 
laboratory  equipment  and  subject-matter  experts  to  act  as  facilitators.  If  the  laboratory  requires 
costly,  complex,  or  difficult  to  maintain  laboratory  equipment  and/or  the  required  facilitator’s  skills 
begin  to  approach  those  of  a  trained  instructor,  the  utility  of  this  strategy  breaks  down. 

A  fourth  strategy  is  to  develop  on-board  training  packages  that  can  be  administered  by  the 
student’s  supervisor  on  the  job  rather  than  in  formal  training.  This  strategy  is  probably  practical, 
though  it  would  transfer  training  burden  to  ship’s  crew  and  reduce  training  quality  control. 

One  may  envision  other  strategies  as  well.  Thus,  a  key  issue  in  assessing  the  feasibility  of 
hands-on  VTT  is  the  choice  of  laboratory  strategy.  The  research  presented  in  this  technical  note 
investigated  two  laboratory  strategies:  (1)  having  students  view  videotapes  of  laboratory 
procedures  instead  of  participating  in  a  laboratory  and  (2)  having  a  facilitator  conduct  the 
laboratories  off-line. 


METHOD 

Overview 

The  method  included  steps  to  develop  the  research  plan,  prepare  for  VTT,  and  collect  and 
analyze  data.1  These  steps  arc  described  below. 

Research  Plan 

The  research  plan  is  described  below  in  terms  of  research  objectives,  research  design, 
dependent  variables,  data  collection  methods  and  instruments,  subjects,  and  data  collection. 

Research  Objectives 

The  objective  of  the  work  was  to  test  the  feasibility  of  using  VTT  to  deliver  hands-on  training. 
Feasibility  is  assessed  in  terms  of  training  effectiveness  and  acceptance  by  students  and  instructors. 
The  baseline  for  comparison  is  traditional  live  instruction.  It  was  not  expected  that  any  VTT 
technology  would  improve  training  effectiveness  or  acceptance;  parity  with  live  instruction  would 
validate  the  technology.  This  objective  was  addressed  by  determining  the  effects  upon  dependent 
variables  of  student  participation  in  (1)  live  versus  VTT  instruction  and  (2)  two  alternative 
laboratory-  strategics. 

A  secondary  objective  was  to  assess  the  relative  training  effectiveness  of  1-way  vi '  o  with  2- 
way  audio  (IV/2A)  vs.  2-way  video  with  2-way  audio  (2V/2A).  The  resulting  research  was  a 


'The  assumptions  sad  simulation  requirements  underlying  the  NTT  laboratory  aad  the  NTT  laboratory  itself  are 
described  in  detail  in  Simpson  tl  al.  (1991). 
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partial  replication,  with  a  hands-on  course,  of  earlier  research  with  lecture-based  training  as 
reported  in  Simpson  et  al.  (1991). 

Research  Design  and  Independent  Variables 

The  research  consisted  of  two  related  experiments. 

Experiment  1.  Experiment  1  used  two  independent  variables  in  a  2  X  2  design.  Independent 
variables  were  classroom  (local  or  remote)  and  type  of  VTT  technology  (1V/2A  or  2V/2A). 

Experiment  2.  Experiment  2  used  a  two-group  design.  The  independent  variable  was  type  of 
laboratory  strategy  (laboratory  or  video  aid).  Subjects  in  the  laboratory  (control)  group  completed 
a  traditional  hands-on  laboratory  and  then  received  a  performance  test  Subjects  in  the  video  aid 
(experimental)  group  viewed  a  videotape  of  lab  procedures,  without  completing  a  hands-on  labo¬ 
ratory,  and  then  received  a  performance  test 

Dependent  Variables 

Dependent  variables  were  student  performance  on  a  written  final  examination,  student 
performance  on  performance  tests,  and  student  attitudes  on  several  different  factors  as  reflected  in 
written  course  evaluations.  The  methods  and  instruments  used  to  collect  these  data  are  described 
below. 

Data  Collection  Methods  and  Instruments 

Student  background  questionnaire:  Students  completed  a  one-page  “Student  Survey’* 
(Appendix  A)  at  the  start  of  each  course.  The  questionnaire  provides  information  on  student 
seniority  and  course  subject-matter  experience. 

Performance  tests:  Performance  tests  were  developed  and  administered  to  a  random  sample 
of  62  students  following  training.  Performance  tests  were  used  as  a  dependent  measure  only  for 
experiment  2.  The  testing  procedure  was  time  and  personnel  intensive,  which  made  rt 
impractical  to  administer  tests  to  all  students.  The  performance  tests  assessed  student  speed  and 
accuracy  in  performing  three  laboratory  tasks:  watertight  door  chalk  test,  marine  strainer 
disassembly/assemfely,  and  door  doc  maintenance.  Damage  Control  Petty  Officer  (DCPO) 
instructors  recommended  that  these  tasks  be  used  for  performance  testing  because  they 
represented  a  perceived  range  of  difficulty  from  chalk  test  (“easy”)  to  door  dogs  to  strainer 
("difficult”).  Performance  tests  were  administered  by  an  instructor  to  individual  students  in  a 
secure  area.  The  performance  testing  procedure,  scoring  rules,  student  instructions,  and 
performance  tests  are  contained  in  Appendix  B.  The  performance  testing  procedure  is 
described  in  greater  detail  in  the  Classroom  Procedures  subsection. 
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Written  final  examinations:  Written  final  examinations  were  administered  to  215  students  at 
the  end  of  the  course.  The  exams  were  taken  “closed  book*’  and  consisted  of  50  4-choice 
multiple-choice  items.  One  form  of  the  final  was  used  throughout  research. 

Student  course  evaluations:  Attitude  measures  were  obtained  using  Likert  rating  scales  on  a 
series  of  questions  relating  to  the  instructor,  audio-visual  aids,  tests/homework,  training  aids, 
labs,  overall  assessment,  and  instructor-student  interaction.  Student  comments  on  the  class 
were  gathered  in  a  series  of  open-ended  questions.  A  sample  evaluation  questionnaire  is 
contained  in  Appendix  C. 

Observations;  Navy  Personnel  Research  and  Development  Center  (NPRDC)  observers  moved 
between  classrooms  during  training  and  maintained  a  written  log  of  significant  class  events. 

Instructor  debriefings:  Instructors  were  debriefed  by  NPRDC  researchers  at  the  end  of  each 
week  and  comments  were  recorded  in  a  written  log. 

Subjects 

Subjects  were  Navy  active  duty  personnel  in  a  variety  of  ratings  undergoing  training  required 
by  their  duty  position.  Ranks  ranged  bom  E-2  to  E-8,  with  the  majority  of  students  E-4  to  E-6; 
mean  rank  was  E-5.  Students  weir  assigned  to  a  classroom  (local  or  remote)  by  instructors,  who 
attempted  to  balance  the  relative  sizes  and  seniority  of  local  and  remote  classes.  Each  classroom 
typically  contained  about  18  students. 

Data  Collection 

Data  were  collected  over  a  6- week  period  beginning  in  June  1991.  Data  were  collected  for  a 
single  2-day  class  during  each  week.  The  first  3  weeks  of  data  collection  simulated  2V/2A  and  the 
final  3  weeks  simulated  1V/2A. 

Preparation  for  VTT 


Training  Course  Selection 


The  research  was  conducted  using  the  Fleet  Training  Center’s  (FTC)  2-day  Divisional  DCPQ 
Indoctrination  course  (J-495-0400).  The  DCPO  course  was  recommended  Oy  Commander, 
Training  Command.  U.S.  Pacific  Fleet  (COMTRAPAC),  with  the  concurrence  of  the  FTC.  It  was 
an  ideal  candidate  for  research  in  VTT  hands-on  training  because  it  is  short,  in  high  demand,  has 
high  student  throughput,  and  uses  class  processes  typical  of  many  other  hands-on  courses.  It 
includes  lectures,  written  tests,  demonstrations  with  various  training  aids,  presentation  of  visual 
aids  (transparencies,  videotapes),  and  laboratories  during  which  students  disassemble,  adjust, 
reassemble,  and  test  damage  control  equipment.  No  performance  testing  is  administered  during 
traditional  DCPO  courses  and  we  had  no  reason  to  expect  the  written  lest  to  highlight  performance 
differences  among  students  as  a  f  unction  of  live  vs.  VTT;  hence,  we  developed  a  performance  test 
(Appendix  B). 
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The  course  differs  from  previous  lecture-based  courses  we  have  conducted  research  with 
primarily  in  terms  of  its  requirement  for  demonstrations  with  many  large  training  aids  (e.gM 
hatches)  and  requirement  for  hands-on  laboratories. 

Classroom  Procedures 

Training  and  the  classroom  itself  had  to  be  adapted  for  VTT.  The  adaptation  procedure 
followed  is  described  in  Simpson  (in  press).  Classroom  procedures  were  constrained  by  the  audio/ 
video  equipment  used,  classroom  design,  and  the  requirement  to  conduct  training  and  manage  two 
classes  simultaneously.  The  classroom  floor  plan  is  shown  in  Figure  1.  The  classroom  was 
equipped  with  two  stationary  instructor  cameras,  one  pointed  at  the  lectern  area  and  the  second  at 
the  demonstration  area  (to  the  instructor’s  left,  when  facing  the  class).  Instructors  had  to  restrict 
their  movement  left  and  right  to  remain  in  the  picture  frame.  The  instructor  was  provided  with  an 
easel  camera,  which  was  used  to  present  visual  aids  and  for  writing  (instead  of  a  writing  board).  A 
video  switch  enabled  the  instructor  to  select  the  camera  or  other  video  source. 


Figure  1.  VTT  classroom  floor  plan  for  damage  control  petty  officer  course. 


No  modifications  were  made  to  the  content  of  lectures,  class  written  materials,  or  written  tests, 
though  performance  tests  were  selectively  used.  VTT  ana  live  course  length  were  identical. 
Viewgraphs  used  in  the  live  class  were  converted  to  hardcopy  form  to  improve  their  appearance  on 
TV  displays. 

Demonstrations  and  laboratories  had  to  be  modified  for  VTT.  The  traditional  live  class  was 
conducted  in  a  classroom  with  many  large,  stationary  training  aids  (e.g.,  vertical  ladder,  various 
hatches,  bulkhead-mounted  marine  strainer)  located  about  the  room.  During  the  class,  the 
instructor  would  move  from  training  aid  to  training  aid  to  conduct  the  demonstration.  Capturing 
the  instructor’s  picture  on  TV  would  have  required  a  dedicated  camera  at  each  demonstration 
position  or  a  camera  operator  with  movable  camera.  Since  both  alternatives  were  impractical,  two 
strategies  were  used  to  support  the  demonstrations:  (1)  the  instructors  fabricated  a  portable,  multi¬ 
purpose  training  aid  that  could  be  rolled  on  and  off  camera  in  the  VTT  classroom’s  demonstration 
area  and  (2)  some  demonstrations  were  videotaped  and  later  presented  to  the  class  instead  of  live 
demonstrations. 

The  most  challenging  aspect  of  hands-on  VTT  was  to  adapt  the  laboratories  for  remote-site 
delivery.  During  traditional  live  courses,  the  instructor  strolled  the  laboratory  stations  and  worked 
closely  with  students  to  demonstrate  procedures,  supervise  students,  assure  safety,  and  certify 
student  performance.  Our  observations  of  laboratories  indicated  that  instructor  demonstrations 
were  usually  incomplete  but  that  students  could  master  laboratory  procedures  anyway  by  working 
with  peers,  relying  on  prior  experience,  or  using  trial  and  error. 

Du;.ng  tfre  research,  the  traditional  laboratory  procedure  was  followed  with  students  in  the 
local  class.  However,  this  could  not  be  done  at  the  remote  class  since  no  instructor  would  be 
physically  present  there.  The  compensatory  strategy  used  in  the  remote  class  had  three  elements: 
(1)  duplicate  laboratory  equipment,  (2)  videotaped  laboratory  demonstrations,  and  (3)  facilitator  to 
supervise  laboratory,  assure  safety,  and  certify  performance.  The  duplicate  laboratory  equipment 
permitted  remote  students  to  gain  hands-on  experience,  the  videotapes  provided  demonstrations  in 
the  absence  of  a  live  instructor,  and  the  facilitator  performed  the  instructor’s  three  remaining 
laboratory  functions.  The  facilitator’s  role  was  defined  as  that  of  a  qualified  DCFO  (i.e.,  a  subject- 
matter  expert)  but  not  an  instructor;  thus,  the  facilitator  answered  questions  but  did  not  engage  in 
formal  instruction  during  the  laboratory. 

Videotapes  were  made  of  the  instructor  performing  laboratory  demonstrations.  The  videotapes 
covered  six  tasks:  (1)  marine  strainer  disassembly/assembly,  (2)  watertight  door  chalk  test,  (3)  door 
dog  maintenance,  (4)  battle  lantern  disassembly/assembly,  (5)  ladder  maintenance,  and  (6) 
watertight  scuttle  maintenance.  The  tapes  were  made  based  on  Personal  Qualification  Standards 
(PQS)  procedures  and  were  accurate  and  complete;  hence,  they  provided  a  more  systematic  and 
comprehensive  demonstration  than  the  typical  live  demonstration.  Each  of  the  videotapes  averaged 
5  minutes  in  length.  They  were  produced  in  a  period  of  approximately  3  weeks  by  a  team  of  three 
DCPO  instructors  and  a  crew  of  audio-visual  specialists  from  NPRDC.  The  instructors  created 
scripts  for  each  video  based  on  PQS  procedures.  They  made  arrangements  to  shoot  the  videos  on 
board  ship  and  in  the  DCPO  classroom.  NPRDC  audio-visual  specialists  made  videotapes  of  the 
instructors  performing  the  procedures  on  camera  and  later  provided  a  VHS  master  allowing  the 
instructors  to  identify  editing  requirements.  The  instructors  then  worked  closely  with  audio-visual 
personnel  during  editing  and  recording  of  a  sound  hack.  In  pan,  this  effort  was  conducted  to  test 
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the  feasibility  of  “grass  roots”  video  production.  The  resulting  videos  were  simple  but  technically 
accurate,  and  were  produced  at  a  fraction  of  the  cost  of  professional  quality  video. 

In  traditional  DCPO  training,  the  instructional  sequence  is  (1)  lectures  and  demonstrations,  (2) 
laboratory,  and  (3)  written  final  examination.  During  experiment  1,  students  in  both  local  and 
remote  classes  followed  this  procedure,  though  laboratories  in  the  local  class  were  overseen  by  an 
instructor  and  those  in  the  remote  class  were  overseen  by  a  facilitator. 

During  experiment  2,  students  in  the  local  class  went  through  the  instructional  sequence  (1) 
lectures  and  demonstrations,  (2)  laboratory,  (3)  performance  test,  and  (4)  written  final  examination. 
Students  in  the  remote  class  went  through  the  instructional  sequence  (1)  lectures  and 
demonstrations,  (2)  videotaped  demonstrations,  (3)  performance  test,  (4)  laboratory,  and  (5) 
written  final  examination.  Note  that  students  in  the  remote  class  were  performance  tested  before 
they  received  any  hands-on  training  in  the  laboratory.  Performance  testing  involved  individual 
testing  of  students  by  an  instructor  and  was  very  time  and  labor  intensive.  For  this  reason,  only  a 
sample  of  students  (62)  was  tested.  Note  also  that  the  written  final  examination  tested  student 
knowledge  of  information  presented  during  lectures  and  demonstrations,  not  laboratories,  so  that 
final  examination  scores  should  not  be  affected  by  the  differing  laboratory  experiences  of  local  and 
remote  classes. 

For  lectures,  traditional  instructor-student  interaction  procedures  were  modified  for  VTT.  The 
instructor  made  seating  charts  of  both  classrooms  and  systematically  alternated  questions  between 
classrooms.  In  all  research  conditions,  students  were  required  to  identify  self  by  name,  pause  to  be 
recognized  by  the  instructor,  and  then  asked  their  question.  The  instructor  would  sometimes  repeat 
a  question  before  answering. 

The  written  final  examination  was  administered  and  scored  in  the  local  classroom  by  the 
instructor  and  in  the  remote  classroom  by  the  facilitator;  results  were  transmitted  from  remote  to 
local  classroom  via  facsimile  machine. 

During  experiment  2,  performance  tests  were  administered  and  scored  by  five  different 
instructors  in  six  different  secure  testing  areas.  Tests  were  administered  by  instructors  who  had  no 
knowledge  of  whether  the  student  was  from  the  laboratory  or  video  aid  group.  All  instructors 
followed  the  performance  testing  protocol  contained  in  Appendix  B. 

Instructor  Training 

Three  different  instructors  delivered  training  during  the  experiment.  Each  instructor  conducted 
two  classes,  with  the  assistance  of  a  second  instructor  playing  the  role  of  a  facilitator  at  the  remote 
classroom.  All  instructors  had  graduated  from  Navy  instructor  training  school  and  were  qualified 
to  teach  damage  control.  NPRDC  research  personnel  worked  closely  with  the  instructors  during  an 
informal  1-weck  training  period.  Instructors  were  familiarized  with  the  audio  and  video  equipment 
and  practiced  equipment  operation  and  class  procedures.  The  total  training  period  per  instructor 
was  approximately  2  days,  most  of  it  devoted  to  practice  teaching.  None  of  the  instructors  was 
given  or  had  previously  received  training  in  camera  presence,  articulation,  graphics  production,  or 
other  skills  of  TV  professionals. 
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VTT  Laboratory 

The  cameras  and  other  equipment  used  in  the  VTT  laboratory  are  described  in  detail  in 
Simpson  et  al.  (1991).  The  classroom  floor  plan  was  shown  earlier  in  Figure  1.  Students  sit  at 
tables,  with  two  chairs  per  table.  Each  table  is  equipped  with  low-profile  sound-activated 
microphone.  Large-size  TV  displays  are  used:  45M  rear  projection  TV  as  main  display,  35"  tube  as 
secondary  display.  Tables  are  arranged  in  amphitheatre  fashion  so  that  all  students  are  seated  within 
a  90-degree  arc  originating  from  the  center  of  the  main  TV  display  to  assure  an  adequate  view.  The 
secondary  TV  display  is  located  to  the  left  of  the  primary  display.  The  primary  display  shows 
outgoing  video  in  the  local  class  and  incoming  video  in  the  remote  class.  The  secondary  display 
shows  students  in  the  other  class. 

Each  classroom  has  four  different  TV  cameras:  (1)  instructor  lectern,  (2)  instructor 
demonstration  area,  (3)  easel  camera,  and  (4)  class.  A  multi-channel  video  switch  is  used  to  select 
which  camera’s  signal  to  send  to  the  other  classroom.  The  instructor’s  cameras  are  suspended 
above  the  second  row  of  tables.  The  class  camera  is  located  above  and  behind  the  main  TV  display. 

The  instructor  wears  a  continuously-on,  wireless,  clip-on  lavaliere  microphone  and  stands 
behind  a  lectern  at  the  front  of  the  classroom  to  the  left  of  the  primary  TV  display.  The  video  switch 
(a  sm.ril  panel  with  seven  push-buttons)  is  attached  to  the  side  of  the  lectern  so  that  the  instructor 
can  r.  -ch  down  and  switch  cameras  with  the  left  hand.  On  the  table  to  the  left  of  the  instructor  are 
air  easel  camera  and  a  TV’  monitor  used  for  orienting  graphics  on  the  easel  camera.  A  25"  TV 
display  on  the  flo^r  before  the  first  row  of  students  faces  the  instructor  and  shows  incoming  video 
f-'om  Jie  remote  class,  each  classroom  is  equipped  with  a  facsimile  machine  and  telephone 
connected  to  other  classioomfs)  via  ring-down  telephone  circuits. 

The  classroom  design  ;s  similar  to  the  one  we  have  used  for  lecture-based  instruction  with  the 
following  exceptions: 

1 .  Open  floor  space  is  provided  nr.  the  left  side  of  Lite  room  for  storage  of  training  aids. 

2.  First  row  of  students  is  moved  back  to  provide  space  for  instructor  to  perform 
demonstrations  with  training  aids. 

3.  Movable  lectern  is  provided. 

4.  Instructor  workstation  is  simplified. 

5.  Instmuoi  is  provided  with  cordless  microphone  (needed  for  additional  mobility). 

RESULTS 


Experiment  1 

Conditions  are  compared  below  based  on  (1)  final  examination,  (2)  course  questionnaire,  and 
(3)  observations. 
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Final  Examination 

A  total  of  215  students  completed  the  final  examination.  Final  examination  scores  as  a  function 
of  classroom  (local  or  remote)  and  VTT  technology  (1V/2A  or  2V/2A)  are  summarized  in  Table  1. 
A  2- way  analysis  of  variance  was  conducted  using  classroom  and  VTT  technology  as  independent 
variables  and  final  examination  score  as  the  dependent  variable.  None  of  the  main  effects  or 
interactions  was  statistically  significant  Student  performance  was  not  significantly  affected  by 
whether  students  were  present  in  the  local  or  remote  classroom.  Test  scores  were  slightly  higher 
with  2V/2A  technology  as  compared  to  1 V/2A,  but  this  difference  was  not  statistically  significant 
either. 


Ifeblel 


Final  Examination  Scores  by  Classroom  (Local  or  Remote)  and 
VTT  Technology  (1V/2A  os*  2V/2A) 

(Numbers  are  percentages) 

(N  =  215) 


VTT  Technology 

Scores  (%)  bv  Classroom 

Local 

Remote 

Overall 

1V/2A 

89.84  (N  =  62) 

89.54  (N  =  65) 

89.69  (N=  127) 

2V/2A 

94.67  (N  =  42) 

90.83  (N  -  46) 

91.23  (N=  88) 

Overall 

90.58  (V=  104) 

90.07  (V=  111) 

90.32  (JV  =  215) 

Experiment  1  is  a  partial  replication  of  work  reported  in  Simpson,  Pugh,  and  Parchman  (1991). 
That  research  investigated  the  relative  training  effectiveness  and  user  acceptance  of  live  instruction 
and  six  different  alternative  VTT  technologies  for  lecture-based  training.  The  present  study 
enabled  the  extension  of  that  research  for  the  lecture  and  discussion  portions  of  a  hands-on  training 
course.  The  earlier  research  indicated  that  student  performance  was  comparable  in  live  classrooms 
and  VTT  classrooms  using  1 V/2A  or  2V/2A. 

Course  Questionnaire 

Student  attitudes  were  measured  with  a  post-course  questionnaire  (Appendix  C)  which 
contained  a  series  of  statements  to  be  rated,  multiple-choice  questions,  and  open-ended  questions. 
Questionnaires  were  completed  by  199  students.  The  results  are  collapsed  across  technologies  (IV/ 
2A  and  2V/2A)  and  described  below  in  terms  of  classroom  (local  or  remote),  the  variable  of 
primary  concern  in  experiment  1.  We  administered  a  closely-related  questionnaire  to  students 
during  previous  research  and  the  results  are  presented  in  Simpson  et  al.  (1990,  1991).  Since  the 
present  findings  closely  match  those  reported  previously,  the  following  discussion  is  brief. 

Student  Ratings.  The  statements  to  be  rated  fell  into  six  categories  (instructor,  audiovisual 
aids,  tests  and  homework,  training  aids,  labs,  and  overall  assessment  of  instructor  and  course). 
Statements  were  rated  on  a  5-poinl  scale  with  a  midpoint  of  3.  Mean  ratings  were  computed  for 
local  and  remote  classrooms  and  are  shown  in  Table  2.  The  majority  of  ratings  on  all  items  fell  well 
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Table  2 


Attitude  Measures  on  Student  Course  Questionnaire  (Scale  1*5) 

(N  =  199) 


Responses*  by  Classroom 

Question 

Local 

Remote 

Local-Remote 

Instructor 

1.  Instructor  prepared  for  class. 

4.4 

4.2 

0.2 

2.  Instructor  presented  lesson  clearly. 

4.4 

4.2 

0.2 

3.  Instructor  answered  student  questions. 

4.5 

4.3 

0.2 

4.  Instructor  encouraged  student  participation. 

4.4 

4.3 

0.1 

Audio-visual  Aids 

5.  Image  on  video  screen  was  large  enough. 

4.0 

42 

-0.2 

6.  Image  on  video  screen  was  clear  enough. 

3.9 

4.1 

-0.2 

7.  Audio  from  other  class  was  loud  enough. 

3.7 

3.5 

0.2 

8.  Audio  from  other  class  was  clear  enough. 

3.6 

3.7 

-0.1 

Tests/Homework 

9.  Test  questions  were  cleady  written. 

4.2 

4.1 

0.1 

10.  Test  questions  related  to  course. 

4.5 

4.3 

0.2 

11.  Test  questions  were  fairly  graded. 

4.5 

4.4 

0.1 

Training  Aids 

12.  Training  aids  supported  instruction. 

4.5 

4.3 

0.2 

13.  Training  aids  were  used  effectively. 

4.5 

4.4 

0.1 

14.  Details  of  training  aids  could  be  seen  clearly. 

4.5 

4.3 

0.2 

Labs 

15.  Labs  helped  me  understand  course  material. 

4.5 

4.5 

0.0 

16.  Labs  helped  me  learn  to  perform  tasks. 

4.5 

4.5 

0.0 

17.  There  were  enough  labs  to  cover  key  topics. 

4.4 

4.4 

0.0 

18.  Job  performance  sheets  were  effective. 

4.2 

4.1 

0.1 

19.  Instructor/facilitalor  provided  help  during  labs. 

4.6 

4.5 

0.1 

20.  Videotapes  during  labs  were  valuable. 

N/A 

4.1 

N/A 

Overall 

21 .  Compare  instructor  with  others  in  past. 

4.0 

4.0 

0.0 

22.  Compare  course  with  others  in  past. 

3.9 

3.7 

0.2 

'Responses  range  from  5  =>  outstanding  to  1  =  unsatisfactory. 


above  the  midpoint  on  the  rating  scale;  most  students  gave  positive  ratings  to  the  dimension  being 
measured.  Differences  between  local  and  remote  classrooms  were  small  and  show  no  interesting 
patterns.  The  main  thing  they  show  is  that,  regardless  of  whether  students  were  in  the  local  or 
remote  classroom,  they  were  positively  disposed  toward  several  different  aspects  of  the  VTT 
learning  environment.  One  possible  problem  area  was  audio  (questions  7, 8),  which  also  parallels 
previous  findings. 
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Figure  2  shows  student  responses  to  question  24.  Most  students  felt  either  that  VTT  had  no 
effect  on  opportunities  to  ask  questions  or  provided  more  opportunities.  However,  in  remote 
classes,  overall,  24.2  percent  of  students  felt  that  VTT  had  provided  fewer  opportunities. 


[H  More  opportunities 
^  No  effect 
H  Fewer  opportunities 


Local  Remote 

Classroom 

Figure  2.  Student  responses  to  question  24  (How 
did  the  VTT  method  of  instruction  affect 
your  opportunities  to  talk  to  the  instructor 
or  ask  questions,  as  compared  to  traditional 
methods  of  instruction?). 

Figure  3  shows  student  responses  to  question  25.  Nearly  one-half  of  students  in  local  and 
remote  classes  expressed  preference  for  traditional  instruction.  Interestingly,  students  in  the  remote 
class  were  about  twice  as  likely  to  prefer  VTT  as  students  in  the  local  class. 

Figure  4  shows  student  responses  to  question  26.  Most  students  either  felt  that  VTT  had  no 
effect  on  learning  or  improved  learning. 

Student  Comments.  Student  comments  were  not  submitted  to  formal  analysis  as  the  research 
was  similar  to  previous  research  we  have  conducted  (Simpson  et  al.,  1990,  1991)  and  would  be 
expected  to  yield  similar  results. 

Observations 

As  in  previous  research,  wc  observed  that  instructors  gained  skill  and  confidence  in  the  VTT 
classroom  rapidly.  All  functioned  effectively  within  a  few  hours,  and  became  comfortable  and 
skilled  before  the  camera  in  about  2  weeks. 

The  processes  occurring  during  laboratories  in  local  and  remote  classrooms  were  similar.  The 
instructor  generally  played  a  more  active  role  in  the  local  classroom  than  did  the  facilitator  in  the 


11 


Response  Rate  (percent)  Response  Rate  (percent) 


Local  Remote 

Classroom 

Figure  3.  Student  responses  to  question  25  (Which 
method  of  instruction  would  you  have  pre¬ 
ferred  for  this  course?). 


□  VTT 
^  Traditional 
■  No  preference 


Local  Remote 

Classroom 

Figure  4.  Student  responses  '.o  question  26  (How  lid 
the  participation  of  student  at  other  sit  is) 
affect  your  learning  during  this  coursed). 


CD  Improved  learning 
HI  No  effect 

Reduced  learning 
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remote.  In  the  local  classroom,  the  instructor  moved  from  laboratory  station  to  station,  performed 
brief  demonstrations,  and  answered  student  questions.  In  the  remote  classroom,  the  facilitator 
moved  from  station  to  station,  answered  questions,  and  tried  to  be  helpful,  but  did  not  conduct 
structured  training.  In  both  classrooms,  the  amount  of  time  the  instructor  or  facilitator  spent  at  any 
station  was  small.  Most  students  had  prior  training  and  experience  performing  the  laboratory  tasks 
and  did  not  need  much  help  from  the  instructor.  Students  appeared  to  rely  more  on  prior  experience 
or  help  from  peers  than  the  instructor/facilitator. 

Experiment  2 

As  noted,  students  in  the  laboratory  group  completed  a  traditional  hands-on  laboratory  and  then 
received  a  performance  test.  Students  in  the  video  aid  group  viewed  a  videotape  of  laboratory 
procedures,  without  completing  a  hands-on  laboratory,  and  then  received  a  performance  test. 

Performance  test  speed  data  are  given  in  Table  3  and  shown  in  graphic  form  in  Figure  5. 
Instructors  had  rated  the  relative  difficulty  of  these  tasks  in  order  from  low  to  high  as  chalk  test, 
door  dogs,  strainer,  the  elapsed  times  parallel  difficulty  predictions.  In  every  case,  students  who 
had  completed  traditional  hands-on  laboratories  took  less  time  to  perform  the  test  than  those  who 
had  only  viewed  videotaped  demonstrations.  Statistical  analysis  results  are  summarized  in  Table  4. 
A  2-way  analysis  of  variance  was  conducted  using  task  (chalk  test,  door  dogs,  or  strainer)  and 
group  (laboratory  or  video  aid)  as  independent  variables  and  elapsed  time  as  the  dependent 
variable.  As  shown  in  Table  4,  task  was  statistically  significant  (F(2,55)=25.75,  p  <  .01).  The 
difference  between  laboratory  and  video  aid  groups  was  statistically  significant  (F(l,55)=4.62,  p  < 
.05).  The  interaction  between  task  and  group  was  not  statistically  significant.  Note  that  the  2-way 
analysis  aggregated  all  tasks  together.  Separate  1-way  analyses  of  variance  were  conducted  for 
each  of  the  tasks,  using  group  as  the  independent  variable.  None  of  the  analyses  yielded  statistically 
significant  results. 


Table  3 

Performance  Test  Speed  Data  in  Seconds  as  a  Function  of 
Group  (Laboratory  or  Video  Aid)  and 
Task  (Chalk  Test,  Door  Dogs,  or  Strainer) 


_ Speed  by  Task 

Chalk  Test  Poor  Dogs  Marine  Strainer 


Group 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Laboratory 

563 

162 

755 

196 

1007 

281 

Video  Aid 

686 

148 

816 

277 

1201 

223 
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1 

Laboratory 


2 

Video  Aid 


GROUP 


Figure  5.  Performance  test  speed  as  a  function  of  group  (laboratory  or 
video  aid)  and  task  (chalk  test,  door  dogs,  or  strainer). 


Ihble  4 

ANOVA  Results  for  Performance  Test  Speed  Data 


Type  of 


Analysis 

Variable 

Of 

F 

P 

2-way 

Task 

2^5 

25.75 

<.01 

2-way 

Group 

1,55 

4.62 

<.05 

2-way 

Group  x  Task  Interaction 

2,55 

<1 

NS 

1-way 

Chalk  Test 

1,21 

3.52 

NS 

1-way 

Door  Dogs 

1.15 

<1 

NS 

1-way 

Strainer 

1.19 

3.04 

NS 

Performance  test  accuracy  data  are  given  in  Table  5  and  shown  in  graphic  form  in  Figure  6. 
Two  aspects  of  these  results  arc  noteworthy  and  both  relate  to  the  door  dogs  task.  First,  the  data 
show  that  task  order  in  terms  of  error  proneness,  from  low  to  high,  is  chalk  test,  strainer,  door  dogs. 
This  is  a  departure  from  the  order  predicted  by  instructors  based  on  task  difficulty  estimates  (chalk 
test,  door  dogs,  strainer).  Second,  we  would  expect  the  laboratory  group  to  perform  better  than  the 
video  aid  group  since  only  the  former  had  hands-on  experience  with  the  tasks  covered  in  the 
performance  tests.  Neither  prediction  is  correct  Statistical  analysis  results  are  summarized  in 
Table  6.  A  2-way  analysis  of  variance  was  conducted  using  task  (chalk  test,  door  dogs,  or  strainer) 
and  group  (laboratory  or  video  aid)  as  independent  variables  and  number  of  errors  as  the  dependent 
variable.  Task  was  statistically  significant  (F(2,56)= 18.94,  p  <  .01).  The  difference  between 
laboratory  and  video  aid  groups  was  not  statistically  significant 

Table  5 

Performance  Test  Accuracy  Data  as  a  Function  of  Group 
(Laboratory  or  Video  Aid)  and  Task 
(Chalk  Test,  Door  Dogs,  or  Strainer) 


Accuracy  by  Task 


Chalk  Test  Door  Dogs  Marine  Strainer 


Group 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Laboratory 

0.44 

0.73 

5.29 

1.38 

1.46 

1.97 

Video  Aid 

1.64 

1.69 

3.24 

2.05 

2.00 

1.63 

Laboratory  _ _  „  Video  Aid 

GROUP 

Figure  6.  Performance  test  accuracy  as  a  function  of  group  (laboratory' 
or  video  aid)  and  task  (chalk  test,  door  dogs,  or  strainer). 
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Table  6 


ANOVA  Results  for  Performance  Test  Accuracy  Data 


Type  of 
Analysis 

Variable 

df 

F 

P 

2-way 

Task 

246 

18.94 

<.01 

2-rvay 

Group 

146 

<1 

NS 

2-way 

Group  x  Task  Interaction 

2,56 

4.86 

<.05 

1-way 

Chalk  Test 

Ul 

3.99 

NS 

1-way 

Door  Dogs 

1,16 

5.37 

<.05 

Strainer 

1,19 

<1 

NS 

There  was  a  statistically  significant  interaction  between  task  and  group  (F(2,56)=4.86,  p  <  .01). 
Group  had  a  different  effect  depending  on  task;  that  is,  the  laboratory  group  did  not  perform  as  well 
as  the  video  aid  group  on  two  tasks  (strainer,  chalk  test),  but  performed  better  on  the  remaining  task 
(door  dogs)  (Figure  6).  The  contrasting  performance  profiles  explain  the  interaction.  Again,  it  is 
the  door  dogs  task  that  underlies  the  effect.  An  explanation  is  needed  to  explain  the  door  dogs 
performance  profile  (see  below). 

Separate  1-way  analyses  of  variance  were  conducted  for  each  of  the  tasks,  using  group  as  the 
independent  variable.  The  only  analysis  yielding  statistically  significant  results  was  for  the  door 
dogs  task  (F(  1,1 6)=5.37,  p  <  .05). 

What  explanations  can  be  offered  for  the  surprising  results  obtained  with  the  door  dogs  task? 
After  obtaining  these  results,  we  discussed  these  tasks  with  the  instructors  and  noted  the  following: 

1 .  Strainer  and  door  dogs  tasks  have  approximately  the  same  number  of  steps. 

2.  Chalk  test  has  approximately  half  as  many  steps  as  strainer/door  dogs  tasks.  It  is  inherently 
a  simpler  task  than  the  other  two. 

3.  Disi  ssembly/assembly  of  the  strainer  is  similar  to  solving  a  3-dimensional  puzzle.  Once  it 
is  disassembled,  there  is  only  one  correct  way  to  reassemble  it  Common  errors  arc  to  omit  non- 
assembly  steps  (e.g.,  lubrication);  there  arc  four  steps  of  this  type. 

4.  Disassembly/assembly  of  the  door  dogs  can  be  done  incorrectly  and  requires  some  finesse. 
Unlike  the  strainer,  it  can  be  reassembled  incorrectly  (e.g.,  by  installing  siring  packing  incorrectly). 
Additionally,  the  door  dogs  task  also  has  many  more  non-assembly  steps  than  the  strainer  (total  of 
9). 

What  is  intriguing  about  the  door  dogs  test  results  is  that  students  in  the  video  aid  group  performed 
better  than  those  in  the  laboratory  group.  Recall  that  students  in  the  video  aid  group  were  tested 
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after  they  had  viewed  a  videotape  demonstrating  the  task  but  before  they  participated  in  a  hands- 
on  laboratory.  Viewing  the  videotape  helped  them  overcome  not  having  participated  in  the 
laboratory  and  actually  gave  them  an  edge  over  students  in  the  laboratory  group  on  the  door  dogs 
task.  The  error  profiles  revealed  that  members  of  the  laboratory  group  were  more  likely  to  omit 
non-assembly  steps  than  members  of  the  video  aid  group.  Assembly  steps  provide  some  feedback 
to  students  on  performance  accuracy  (e.g.,  if  the  part  fits,  it  is  reasonable  to  assume  that  it  has  been 
assembled  correctly).  However,  non-assembly  steps  such  as  lubrication  do  not  provide  such 
feedback.  Hence,  one  possible  rationale  for  the  effect  the  videotapes  had  with  the  door  dogs  task  is 
that  it  was  more  effective  than  live  demonstrations  in  helping  students  learn  non-assembly  steps. 

DISCUSSION  AND  CONCLUSIONS 

The  primary  objective  of  the  research  was  to  test  the  feasibility  of  using  VTT  to  deliver  hands- 
on  training.  Was  feasibility  demonstrated?  The  answer  is,  yes  and  no.  Some  aspects  of  the 
experiment  succeeded  and  some  did  not.  In  addition,  there  is  much  about  hands-on  VTT  that 
remains  unknown. 

VTT  was  effective  for  lecture,  discussion,  and  hands-on  demonstration  portions  of  training  as 
indicated  by  the  final  examination,  student  course  evaluations,  and  observations.  The 
demonstrations  were  more  elaborate  and  demanding  of  both  instructor  and  students  than  those 
given  in  lecture-based  courses  so  this  success  is  significant. 

VTT  classroom  design  (Figure  1)  was  effective  for  hands-on  training.  This  design  was  adapted 
from  the  design  used  for  lecture-based  VTT  by  providing  additional  storage  space,  moving  back 
the  first  row  of  students  to  provide  a  demonstration  area,  simplifying  the  instructor’s  workstation, 
and  providing  the  instructor  with  a  cordless  microphone.  The  design  may  serve  as  a  model  for  use 
by  others  designing  VTT  classrooms  for  hands-on  training. 

As  noted  in  the  Method  section,  the  most  difficult  aspect  of  hands-on  VTT  is  the  laboratory, 
during  which  the  instructor  typically  works  closely  with  students  to  demonstrate  procedures, 
supervise,  assure  safety,  and  certify  student  performance.  Four  different  strategies  for  handling 
laboratories  were  mentioned:  (1)  elimination  of  laboratory,  (2)  instructor  remote  presence,  (3) 
facilitator-conducted  laboratory,  and  (4)  conducting  training  on  the  job  instead  of  in  formal 
training.  The  research  investigated  two  of  th«*sc  strategies  (1, 3).  A  modified  version  of  strategy  1 
(no  laboratory,  but  videotape)  was  less  than  successful.  Students  who  observed  videotapes  instead 
of  participating  in  a  laboratory  took  longer  to  perform  on  all  performance  test  tasks  and  performed 
less  accurately  on  two  out  of  three  tasks.  Viewing  a  videotape  is  no  substitute  for  participating  in 
a  laboratory,  though  it  is  probably  better  than  no  laboratory  at  all. 

The  second  strategy  (remote  laboratory)  appeared  to  be  more  successful,  though  no 
performance  data  were  collected  to  verify  this  to  be  the  case.  Our  observations  indicated  that  the 
learning  processes  occurring  in  remote  laboratories  were  very  similar  to  those  of  a  live  class,  and 
relied  heavily  on  student  prior  knowledge  and  peer  training.  The  laboratories  might  not  have  been 
as  successful  if  students  were  performing  laboratory  tasks  for  the  first  time.  Additional  research  is 
requited  to  determine  whether  oi  not  this  strategy  would  be  generally  effective. 
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The  research  provided  some  evidence  that  videotapes  are  more  effective  than  live 
demonstrations  in  helping  students  learn  to  perform  procedural  steps  that  do  not  in  themselves 
provide  feedback  on  their  correctness  (e.g.,  the  non-assembly  steps  of  the  door  dogs  task).  This 
finding  is  suggestive  and  warrants  further  investigation  on  a  broader  range  of  tasks  than  were 
investigated  in  this  study.  Note  that  it  has  implications  not  only  for  VTT  but  for  any  training 
involving  demonstrations. 

In  converting  hands-on  courses  for  VTT,  students  must  be  provided  with  hands-on  training  in 
some  form.  The  two  most  realistic  strategies  at  present  appear  to  be  strategies  3  or  4.  Additional 
research  should  be  conducted  to  refine  the  laboratory  strategies  used,  to  apply  them  to  other  types 
of  training  (e  g.,  electronics  troubleshooting,  fiber  optics  maintenance,  etc.),  and  to  determine  if 
other  strategies  can  also  be  effective  (e.g.,  strategy  4). 

RECOMMENDATIONS 

1.  The  Chief  of  Naval  Education  and  Training  should  ensure  that  VTT  classrooms  used  to 
originate  courses  with  a  significant  hands-on  training  component  are  designed  to  satisfy  the  special 
requirements  of  these  courses.  Among  the  requirements  are  additional  storage  space  for  training 
aids,  an  open  and  flexible  floor  plan,  a  demonstration  area  covered  by  a  separate  camera,  an 
instructor’s  workstation,  and  a  cordless  microphone  to  provide  the  instructor  with  mobility. 

2.  The  Chief  of  Naval  Education  and  Training  should  employ  the  most  practical  and  effective 
strategy  known  at  this  time  for  conducting  hands-on  laboratories  at  remote  sites:  equip  remote  sites 
with  laboratory  equipment  and  have  a  facilitator  conduct  the  laboratories  off  line.  While  this 
laboratory  strategy  is  workable,  other  less  costly  strategies  should  be  investigated  further. 

3.  The  Chief  of  Naval  Education  and  Training  should  originate  a  Problem  Description  and 
Need  Justification  for  research  to  test  and  evaluate  alternative  strategies  for  conducting  VTT 
hands-on  laboratories.  Among  these  strategies  are  remote  ^  “csence.  the  use  of  on-board  training 
packages  to  allow  supervisors  to  train  students  on  the  job,  and  other  strategies  that  may  be  feasible 
and  effective. 
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APPENDIX  A 

STUDENT  BACKGROUND  QUESTIONNAIRE 


A-0 


STUDENT  SURVEY 

DAMAGE  CONTROL  PETTY  OFFICER 
J-495-Q400 


Date:  Rate  '  Room  No._ , 

Name_ _ _ _  SSN(OPT)_^ _ 

PLEASE  CIRCLE  THE  LETTER  FOR  THE  MOST  APPROPRIATE  ANSWER 


1.  Did  you  request  this  course  of  instruction? 


A. 

Yes  B.  No 

2. 

Have 

you  attended  this  course  within  the  past  2  years? 

A. 

Yes  B.  No  If 

yes,  where? 

3. 

Have 

you  had  any  previous  Damage 

Control  experience? 

A. 

Yes  B ..  No 

4. 

What 

is  your  current  DCPO  PQS  qualification  level? 

a . 

Maintenance  Person 

b. 

Work  Center  Supervisor 

c . 

Division  Officer 

d. 

Departmental  Assistant 

e . 

Department  Head 

f . 

DC  Petty  Officer 

9- 

None 

5. 

What 

is  your  current  assignment? 

a . 

Maintenance  Person 

d. 

Department  Head 

b. 

Work  Center  Supervisor 

e . 

DCPO 

c 

Division  Officer 

f . 

Departmental  DC  Assistant 

6. 

What 

position  will  you  be 

going 

to  upon  completion  of  this 

course? 

a. 

Maintenance  Person 

d. 

Department  Head 

b . 

Work  Center  Supervisor 

e . 

DCPO 

c 

Division  Officer 

f . 

Departmental  DC  Assistant 

FOR  OFFICE  USE  ONLY 
FINAL  GRADE 
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APPENDIX  B 

PERFORMANCE  TESTING  PROCEDURE,  SCORING  RULES, 
STUDENT  INSTRUCTIONS,  AND  PERFORMANCE  TESTS 
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PERFORMANCE  TESTING  PROCEDURE 

Materials 

1.  PMS  card 

2.  Equipment 
Preparation 

Before  testing,  prepare  equipment  by  assembling  or  returning  it  to  pre-test  condition. 

Testing  Procedure 

1.  When  student  enters  testing  area,  introduce  yourself  and  read  “Student  Instructions.” 

2.  Start  stopwatch. 

3.  Observe  student  and  score  performance  on  protocol. 

4.  If  the  time  limit  for  the  test  has  passed,  terminate  test  Otherwise,  when  student  indicates  he/ 
she  is  finished,  stop  stopwatch. 

5.  Record  elapsed  time  on  protocol. 

6.  Caution  student  not  to  discuss  test  with  other  students. 


7.  Excuse  student. 


PERFORMANCE  TEST  SCORING  RULES 


Completeness 

Put  check  mark  by  task  step  if  student  performs  it;  otherwise  leave  blank. 

Errors 

An  error  is  an  attempt  to  perform  a  step  in  an  incorrect  manner.  Examples; 

1.  Not  performing  the  task  to  its  required  standard  (e.g.,  miscalibration). 

2.  Performing  a  task  out  of  sequence  (scored  as  error  if  it  must  be  redone  to  perform  subse¬ 
quent  steps  properly);  ignore  sequence  if  it  has  no  subsequent  effect. 

3.  Attempting  to  assemble  a  part  improperly  (e.g.,  in  reverse  of  its  correct  orientation). 

4.  Failing  to  perform  a  step  completely  (e.g.,  neglecting  to  grease  a  part  [ignore  if  this  would 
be  captured  by  “completeness”  score]). 

5.  Leaving  out  a  part  or  attempting  to  install  an  inappropriate  part. 

Elapsed  Time 

The  amount  of  time  from  start  to  end  of  test.  Stop  test  if  elapsed  time  exceeds  20  minutes. 
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STUDENT  INSTRUCTIONS 


Door  Dogs 

(Read  to  student  before  starting  test) 

You  will  be  taking  a  performance  test.  This  test  is  designed  to  nnd  out  how  accurately  and 
quickly  you  can  perform  the  PMS  procedures  for  the  Watertight  Door  Dogs. 

This  test  will  not  affect  your  grade  in  the  course.  However,  it  is  important  for  you  to  do  your 
best  on  the  test  because  the  results  will  affect  how  training  is  administered  to  other  Navy  personnel 
in  the  future. 

Perform  the  task  as  accurately  and  quickly  as  possible.  I  will  time  how  long  it  takes  you  and 
will  record  your  errors.  The  maximum  amount  of  time  allowed  for  the  test  is  20  minutes. 

The  DCPO  has  told  you  to  check  the  Watertight  door  dogs  for  binding  and  to  disassemble, 
clean,  inspect  and  re-assemble  the  dogs  according  to  correct  procedures. 

I  cannot  talk  to  you,  answer  questions,  or  provide  you  with  help  during  the  test.  Do  you  have 
any  questions  before  you  start?  (Answer  any  non-technical  questions.) 

Are  you  ready  to  begin?  (Start  test  when  student  indicates  readiness.) 
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STUDENT  INSTRUCTIONS 


Watertight  Door  Chalk  Test 
(Read  to  student  before  starting  test) 

You  will  be  taking  a  performance  test.  This  test  is  designed  to  find  out  how  accurately  and 
quickly  you  can  perform  the  PMS  procedures  for  the  watertight  door  chalk  test. 

This  test  will  not  affect  your  grade  in  the  course.  However,  it  is  important  for  you  to  do  your 
best  on  the  test  because  the  results  will  affect  how  training  is  administered  to  other  Navy  personnel 
in  the  future. 

Perform  the  task  as  accurately  and  quickly  as  possible.  I  will  time  how  long  it  takes  you  and 
will  record  your  errors.  The  maximum  amount  of  time  allowed  for  the  test  is  20  minutes. 

The  DCPO  has  told  you  to  perform  the  watertight  door  chalk  test. 

I  cannot  talk  to  you,  answer  questions,  or  provide  you  with  help  during  the  test.  Do  you  have 
any  questions  before  you  start?  (Answer  any  non-technical  questions.) 

Are  you  ready  to  begin?  (Start  test  when  student  indicates  readiness.) 
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STUDENT  INSTRUCTIONS 


Marine  Strainer 

(Read  to  student  before  starting  test) 

You  will  be  taking  a  performance  test.  This  test  is  designed  to  find  out  how  accurately  and 
quickly  you  can  perform  the  PMS  procedures  for  the  Marine  Strainer. 

This  test  will  not  affect  your  grade  in  the  course.  However,  it  is  important  for  you  to  do  your 
best  on  the  test  because  the  results  will  affect  how  training  is  administered  to  other  Navy  personnel 
in  the  future. 

Perform  the  task  as  accurately  and  quickly  as  possible.  I  will  time  how  long  it  takes  you  and 
will  record  your  errors.  The  maximum  amount  of  time  allowed  for  the  test  is  30  minutes. 

The  DCPO  has  told  you  to  disassemble,  clean,  inspect,  and  re-assemble  the  Marine  Strainer  and 
Ball  Valve  assembly. 

I  cannot  talk  to  you,  answer  questions,  or  provide  you  with  help  during  the  test.  Do  you  have 
any  questions  before  you  start?  (Answer  any  non-technical  questions.) 

Are  you  ready  to  begin?  (Start  test  when  student  indicates  readiness.) 
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DC  PETTY  OFFICER  PERFORMANCE  TEST 


WATERTIGHT  DOOR  CHALK  TEST 


Trainees  name: _ Rate: _ 

Time  to  complete: _ minutes : seconds 

Did  trainee  follow  MRC  card?  Yes _  No _ 


Completed 

V 

Errors 

vvv 

Steps 

Clean  gasket  &  apply  chalk  to  knife  edge 

Close  &  dog  watertight  door 

Open  door  &  inspect  chalk  line  on  door  gasket 

Adjust  Affected  dogs 

Clean  gasket  &  chalk  knife  edge 

Close  &  dog  door 

Open  door  &  inspect  chalk  line  on  gasket 

If  line  is  complete ,  clean  gasket,  else  return  and 
readjust  dogs 

Lubricate  gasket  with  light  coat  of  silicone 

Dog  watertight  door  and  inspect  for  tightness 

'.'■'"p..  a  •  a  sBSBSgsasaBss  sassaassasss  sssas _ asess  ■  ~  csssssssssb  .... . sa— ta 
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DC  PETTY  OFFICER  PERFORMANCE  TEST 


MARINE  STRAINER 


Trainees  name: _ 

Time  to  complete: _ 

Did  trainee  follow  MRC  card? 


_ Rate : _ 

_ minutes : seconds 

Yes _  No _ 


Errors 

VV  V 


Remove  eight  bolt  flange 


Remove  and  inspect  gasket 


Remove  and  inspect  ball 


Remove  quick  acting  flush  valve  handle 


Loo<~e  .  six  bolt  flange  to  bell  body  and  break  seal 


Remove  handle  locking  clip  plate 


Remove  bell  body 


Remove  and  inspect  strainer 


Inspect  studs  for  clean  threads 


Remove  bearing  race  access  screw 


Apply  3  dr  )ps  of  light  oil  in  slot  and  replace  screw 


Inspect  hose  gasket  at  coupling  connection 


Reinstall  strainer 


Reinstall  six  bolt  flange  gasket 


Align  and  reinstall  bell  body 


Reinstall  handle  locking  clip  plate 


Reinstall  all  nuts  and  tighten 


Reinstall  T-shaft  from  strainer  handle  to  ball  valve 


Reinstall  quick  acting  flush  valve  handle 


Reinstall  ball  valve 


Reinstall  eight  flange  gasket 


Reinstall  eight  flange  and  nuts 


Check  for  smooth  operation 


DC  PETTY  OFFICER  PERFORMANCE  TEST 


WATERTIGHT  DOOR  DOGS 


Trainees  name: _ 

Time  to  complete: _ 

Did  trainee  follow  MRC  card? 


_ Rate : _ 

_ minutes : seconds 

Yes _  No _ 


COMPLETED 

V 

ERRORS 

V  V  V 

STEPS 

Remove  plunger 

Remove  locking  &  adjusting  nut 

Remove  dog  handle 

Remove  spindle  assembly  from  door  frame 

Remove  spring  &  string  packing  from  spindle 

Remove  inner  bushing  from  spindle 

Remove  flange  bushing  from  door  frame 

Clean  housing  in  door  frame 

Remove  packing  &  clean  spindle 

Clean  the  flange  bushing,  inner  bushing,  &  spring 

Apply  light  coat  of  grease  to  spindle 

Apply  light  coat  of  grease  to  door  housing 

Apply  light  coat  of  grease  to  bushing 

Apply  light  coat  of  grease  to  flange  bushing 

Reinstall  inner  bushing  on  spindle 

Reinstall  spring  on  spindle 

Install  new  string  packing  on  spindle 

Reinstall  flange  bushing  &  set  screw 

Resinstall  spindle  assembly  inside  dog  handle 

Reinstall  adjusting  &  locking  nuts 

Cycle  dog  handle  for  smooth  operation 

Apply  grease  stick  packing 

Insert  set  screw  &  tighten  plunger 

Cycle  dog  handle  for  smooth  operation 
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Course  Questionnaire  for  VTT  Students 


1.  Name 

&  today's  date _ 

Pint 

/  / 

M 

Lott 

Pate 

On  the  last  page,  there  Is  room  to  comment  on  specific  areas  of  concern 
Please  use  this  space  to  clarify  how  An provement  can  be  made. _ 


Section  1:  Course  Evaluation 


For  each  of  the  following  statements  (1  through  2 1).  check  the  appropriate  box  corresponding  to  a  scale  of 

1  (unsatisfactory)  through  S  (outstanding),  with  3  being  average.  Leave  any  statements  that  do  not  apply  to 
this  course  blank. 

.  t  ,  Unsatisfactory - ►  Outstanding 

Instructor  (Average  »  3) 

1 .  Instructor  was  prepared  for  class . 

..  »□ 

.□ 

G 

g 

G 

2.  Instructor  presented  lessons  clearly . 

...  ,□ 

>□ 

3D 

g 

G 

3.  Instructor  answered  student  questions . 

..  ,□ 

d 

a 

g 

G 

4.  Instructor  encouraged  class  participation . 

Audio-Visual  Aids 

..  .□ 

d 

3n 

g 

G 

5.  Image  on  video  screen  was  large  enough  to  be  seen . 

...  n 

d 

n 

g 

G 

6.  Image  on  video  screen  was  clear  enough  to  be  seen . 

..  ,□ 

n 

g 

g 

G 

7.  Audio  from  other  class  was  loud  enough  to  understand . 

..  .□ 

>□ 

a 

d 

d 

8.  Audio  from  the  other  class  was  clear  enough  to  understand . 

Tests/Homowork 

..  a 

d 

3D 

d 

G 

9.  Test  questions  were  clearly  written . 

..  .□ 

»□ 

a 

■a 

a 

10.  Test  questions  were  directly  related  to  course . 

...  .□ 

d 

g 

d 

G 

1 1 .  Test  answers  were  graded  fairly . 

Training  Aids 

.□ 

d 

g 

G 

G 

12.  Training  aids  were  valuable  in  supporting  instruction . 

..  .□ 

d 

a 

G 

d 

13.  Training  aids  were  used  effectively . 

..  ,□ 

2D 

a 

-n 

G 

14.  Details  of  training  aids  could  be  seen  clearly . 

Labs 

..  .□ 

d 

a 

d 

G 

15.  Labs  helped  me  understand  the  course  material . 

...  .□ 

2D 

g 

G 

d 

16.  Labs  helped  me  learn  to  perform  the  tasks  they  covered . 

...  ,□ 

d 

a 

«D 

d 

1 7.  There  were  enough  lobs  to  cover  the  key  topics  in  the  course . 

....  ,□ 

g 

a 

*D 

d 

18.  The  job  performance  sheets  were  effective  in  describing 

task  steps  to  perform  during  the  labs . 

...  ,□ 

,d 

d 

G 

d 

19.  The  instructor  of  ficaStor  provided  enough  help  during  the  tabs. . 

...  ,□ 

»□ 

a 

<Q 

d 

20.  The  videotapes  running  during  the  labs  were  valuable . 

Overall 

..  ,□ 

iD 

g 

a 

d 

2 1 .  Companion  o'  this  instructor  to  other  Navy  instructors  that  hove 

taught  you  in  the  post .  i  □  »□  a  a  d 


22.  Comparison  of  this  course  to  other  Navy  courses  that  you  have  __  __ 

token  in  the  past .  i  D  2D  O  <G  sD 


Section  2:  Instructor-Student  Interaction 


23.  Did  you  talk  to  the  instructor  or  ask  any  questions  during  the  regular  hours  of  this  course? 

f~]  a.  Yes 
D  b.  No 

24.  How  did  the  video  tele-training  method  of  instruction  affect  your  opportunities  to  talk  to  the  Instructor  or 
ask  questions,  as  compared  to  traditional  methods  of  instruction? 

[~]  a.  More  opportunities 

1~~1  b.  No  effect  on  opportunities 

f~~]  c.  Fewer  opportunities 

25.  Which  method  of  instruction  would  you  have  preferred  for  this  course? 

I  I  a.Video tele-training 

n  b.  Live  Instruction  with  instructor  physically  present  In  the  classroom 
I  I  c.  No  preference  between  video  tele -training  and  live  Instruction 

26.  How  did  the  participation  of  students  at  other  srreCs)  affect  your  learning  during  this  course? 

I  I  a.  Improved  learning 
n  b.  No  effect  on  learning 
0  c.  Reduced  learning 

Section  3:  Student  Comments 

27.  What  did  you  like  most  about  this  course? 


28.  What  did  you  like  least  about  this  course? 
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Section  3:  Student  Comments  (continued) 

29.  Discuss  any  suggestions  that  you  have  for  improving  how  video  tele-training  is  used  in  this  course: 


-A  •'» 


■>.  ’I 


30.  Comment  on  usefulness  and  adequacy  of  the  course  content: 


31.  Comment  on  Instructor  preparedness  and  presentation:. 


32.  Comment  on  adequacy  of  training  aids: 


33.  Comment  on  appropriateness  of  the  exams: 


Section  4:  Safety 


34.  Did  lessons  Include  safety  where  applicable  (Safety  os  applies  to  your  job)?. 

1~1  Yes  []  No  Q  See  Remarks 

35.  Did  Instructor(s)  adequately  cover  safety  items  prior  to  conducting  performance  labs? 

f"3  Yes  []  No  Q  See  Remarks 

36.  Was  safety  a  primary  consideration  of  the  Instructor(s)? 

[  !  Yes  []  No  Q  See  Remarks 

37.  Was  the  classroom/laboratory  equipment  a\vays  safe  for  use? 

1  1  yes  Q  Wo  □  See  Remarks 

Comment  on  safety. _ 
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